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In this paper, we develop a general theory for the estimation of the transition probabilities of 
, reversible Markov chains using the maximum entropy principle. A broad range of physical models 

can be studied within this approach. We use one-dimensional classical spin systems to illustrate the 
theoretical ideas. The examples studied in this paper are: the Ising model, the Potts model and the 
£N| , Blume-Emery-Grimths model. 

> 

o 
in 

(N 



X3 



I. INTRODUCTION 



Usually, the only knowledge about a physical system is the measurement of the average values of a few relevant 
observables. The maximum entropy principle [J^El is a tool to obtain the least biased distribution for the equilibrium 
^3 ' distribution of the system that reproduces these measurements. The rough line of reasoning of this approach is as 
follows Q. The Boltzmann-Gibbs entropy functional of a distribution P(j) is defined by 



+i: S = -5>(j)lnP(j), (1) 

> . 

where T is the (discrete) phase space. This entropy functional is most often used in statistical mechanics. The 
d • equilibrium distribution is obtained by maximising the entropy functional under the constraints that the average 
values of the relevant observables H\(j), . . . , H t (j) take on certain values. To solve this optimisation problem, usually 
I . the method of Lagrange multipliers is used. For every constraint, a Lagrange multiplier 0j is introduced and the 
following function is maximised 

o 

ii; £ = S-5>^P(j)^(j), (2) 

i=l j£T 

CO 1 

^ ' to obtain the equilibrium distribution. After variation with respect to P(j) one obtains 

as : 

P(j) ~ expf-V^O')) . (3) 

co ; V i=i / 

With this distribution, one can calculate expressions for the equilibrium values (Hi(j)) of the observables as a function 
' of the Lagrange multipliers 9i. These relations can then be used to estimate the values of 9i after measurement of 
(Hi{j)). In this way, one obtains the least biased estimate of the parameters 9i, because P(J) satisfies the maximum 
entropy principle. The standard example of this procedure is the estimation of the temperature by the measurement 
. ,— i \ of the energy. 

The maximum entropy principle can be used to introduce thermodynamic parameters in simple theoretical models 
[S B • One starts from a mathematical model and calculates the entropy and the average of the relevant observables 
as a function of the model parameters. Then the maximisation procedure is carried out over the model parameters only, 
instead of over all the possible probability distributions. The usefulness of this approach is already shown for specific 
models containing 2 and 5 parameters, see 0, Q and [3] respectively. In this paper we show that the maximisation 
procedure can be carried out for reversible iV-state Markov chains. This problem of conditional optimisation is more 
general and is also studied in the information theory framework [1, [§] . However, we focus on the applications of this 
technique in the context of statistical mechanics and use the maximum entropy principle to relate microscopic and 
macroscopic quantities of physical models. To illustrate the theoretical ideas, one-dimensional classical s pin systems 
are studied, the Ising model the Potts model [ll|, [l2j and the Blume-Emery-Griffiths model [Lsl. flij. These 
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examples serve to show that our theoretical procedure is very general and that a broad range of physical models can 
be studied within this approach. It is not the aim of the present paper to make progress in the understanding of the 
aforementioned models. 

The outline of the paper is as follows. In the next section we fix our notation and repeat briefly some results 
obtained in [l5| that will be used throughout the paper. The basic idea of this work is introduced in section IIIII 
with the use of a simple example, the 2-state Markov chain. The main result of this paper is obtained in section ITVl 
in which we apply the maximum entropy principle to the reversible iV-state Markov chain. In section [V] we make 
the connection between our theory and thermodynamics and examine under which conditions our technique coincides 
with the standard approach to introduce thermodynamic parameters in theoretical models. In sectionfVH the example 
of the 3-state Markov chain is thoroughly studied. The final section contains a summary of our results and a brief 
discussion of the different assumptions we made throughout the paper. 



II. NOTATION 



We consider a finite state space T with N states. A Markov chain is defined by initial probabilities p(z) and 
transition probabilities w{y, z), with y, z S T. The equation of motion is simply 

Pt+i(z) = ^2pt(y)w(y,z), with p (y) =p(y)- (4) 
j/er 

Throughout the paper, we will assume that w{z, y) ^ for all z, y £ T. This means that we study irreducible Markov 
chains [IB]. The N initial probabilities p(z) and the N 2 transition probabilities w(z,y) can be interpreted as the 
parameters of the Markovian model. However, they are not independent because of the normalisation conditions 

1 = £p(y), (5) 
1 = VzeT. (6) 

As a consequence, the Markovian model contains only (N — 1) + (A 2 — N) = N 2 — 1 independent parameters. Usually, 
extra restrictions on the parameters are assumed. A Markov chain is called stationary [l6|, when the following 
condition holds 

p( z ) = ^2p(y)w(y, z ), Vz e r. (7) 

This is a set of N — 1 extra equations (the normalisation is already taken into account). A stronger constraint is 
detailed balance 



p{z)w{z, y) = p{y)w(y, z), Vz, y £ I\ 



(8) 



This is a set of N(N — l)/2 extra equations. Throughout the paper, we will assume that this condition is satisfied. 
This means that we study reversible Markov chains [161 ]. 

A path 7 = (xq, Xi, X2 ■ ■ ■ , x n ) of the Markov chain with length n + 1 has probability 



p(x )w(x ,xi) . . .w(x n -i,x n ). 



(9) 



In [la ] the record of transitions k is introduced. This is a sequence of numbers k(z,y), one for each pair of states 
z, y G r, counting how many times the transition from z to y is contained in a give n path of the Markov chain. The 
ensemble average of the elements of the transition records and the entropy S (17L [l8| of the Markov chain can be 
calculated as follows: 



(k(z,y)) = . . . p(x )w(xq,xi) . . . w(x n -i,x n ) 

xoET x n £F 



E 5 - 

i=Q 



z5 Xi + 1 ,y 



S = - E • ■ ■ E P( x o) w ( x o> x i) ■ 
x er i„er 



.w(x n - 1 ,x n )]n\p(x )w(x ,x 1 ) . ..w(x n _x,x n )] 
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with 8i j the Kronecker delta. For stationary Markov chains, these expressions simplify to [15| 

-( k ( z ,y)) = p( z ) w ( z ,y), (io) 

n 

-S = -^2^2p(z)w(z,y)\nw(z,y) ^p(z)\np(z). (11) 

zeryer zer 

The second term in the expression of the entropy is unimportant for large chains and will be ignored in the remaining 
part of this paper. The technical consequences of taking these finite size effects into account are already thoroughly 
studied for the 2-state Markov chain [slfiiil]. 

The conditional probability P(k;xo) to observe a Markov chain with certain transition record k given the initial 
condition xq G T is 

P(k; xo) = c(k; x ) ]\ II w & f)*^ = x ^ ex P ( E E fc fo v) ^{x, y) ) , (12) 
server \seryer 

where the prior probability c(k; xq) counts the number of paths that have the same transition record k. Notice that the 
value of c(k;xo) can vanish. An obvious example is a combination of an initial condition xq with a transition record 
in which no transition xo — > x with x G T occurs. However, this is not the only possibility to obtain c(fc;xo) = 0. To 
see this, observe that there are two ways to count the number of occurrences of a state i£T given k and xq G T 

3x,x + ^ ky,x and S x , Xn + ^ kx.y, (13) 
yer y er 

where the Kronecker deltas 8 x ,x and 5 XjXn take into account the first and last state of the path respectively. Given 
k and £o G T, only when following equality 

3x,x + ^ * ky-x = &x,x n "F ^ ' ^an,y; ^ x £ I\ (14) 

yer y er 

is fulfilled, one ends up with a unique value for the number of occurrences of every state ier. Therefore, expression 
(I14[) . is a necessary condition in order to obtain a non-vanishing value for c(k;xo). This shows that the elements of 
the transition record are not independent. In section [Vl the importance of this observation will become clear. 

A Markov chain can be interpreted as a sequence of letters where the transition record k counts the number of 
occurrences of two-letter words. Markov chains with a finite memory and generalisations of k are examined in the 
information theory framework [13, HH and find applications in, e.g., the computational biology [13, H3l- l n * ne present 
paper we study Markov chains in the context of statistical mechanics and apply our results to physical models with 
only nearest neighbor interactions. The notion of two-letter words is sufficient for these applications. The extension 
of our theoretical results to systems with next (or higher order) nearest neighbor interactions is merely technical and 
can be obtained by increasing the number of states of the chain which allows to maintain the Markov property. 

III. EXAMPLE: THE 2-STATE MARKOV CHAIN 

In this section we study a simple example, the 2-state Markov chain. The two states are denoted + and — . The 
different parameters of the Markovian model are 

p(+),p(-) and w(+,+),w{+,-),w(-,-),w(-,+). (15) 

However, the number of independent parameters is reduced by 3 because of the normalisation conditions ([5]), ©• 
The detailed balance condition {5} further reduces this number by 1. We conclude that this simple microscopic model 
contains only 2 independent parameters. We chose w(+, — ) and w(— , +) to be these parameters and use (0, ([6]) and 
© to obtain the following relations 

P (+)= , w {-> + ] -, p(-)- w{+ '- ] 



«>(-,+)+ «;(+,-)' «;(-,+) +«;(+,-)' 

«;(+,+) = 1 -w(+,-), ifl(-,-) = l-w(-,+). (16) 
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In section [TTI we introduced the transition record k. The matrix k contains only 4 elements for this example 

(17) 



k(+,+) k(+,-) 
k(-,+) fe(- -) 



The 2-state Markov chain can be interpreted as a one-dimensional Ising chain [T(| with constant length n + 1 and two 
different spin- values ±1. Two relevant observables are 

n— 1 n 

Hi(cr) = - J ^CiCTi+i and ^(c) = X! ' 4 ' ( 18 ) 

with J a constant. The spin variables Uj are scalars that can take on the values ±1. The two states of the Markov 
chain +, — correspond to the spin values +1,-1 respectively. Therefore, one can express H\{a) and i?2(f) as a 
function of the elements of the transition record k as follows: 

fli(fc) = -J[k(+,+) + k(-,-)] + J[fc(+, -) + *(-,+)], (19) 
# 2 (fc) = *(+,+) + *(-,+)- *:(-,-) -*;(+,-). (20) 

The correspondence between -Hi(cr) and H\(k) is exact, while we ignored the contribution of the initial spin cto to 
obtain H2(k) from ^(c)- This is only a finite size effect that is unimportant for large chains. This means that the 
correspondence between H 2 {(j) and H-2(k) is also exact for infinite chains. With (JTUJ) and (fTB|) . one can immediately 
write out the ensemble averages of these variables as a function of the independent parameters w(+, — ) and w(—, +) 

(Hi(k)) = Aw(-,+)w(+, -) - w(-, +) - w(+, -) 

Jn w(—,+) + w(+,—) 

(H 2 {k)) = w(-,+)-w(+,-) 

n w(—,+) + w(+,—) 

Also the entropy (TTTj) of the Markov chain can be expressed as a function of w(+, — ) and w(— , +) only 

! = -^F^1t^F-) ([1 ln[1 -«(+.-)] +«(+,-) ]»»(+-)) 

+w{+,-) 

We use (Hi(k)) and (H2(k)) as constraints in the maximisation procedure ([2]) 

C = S-e 1 (H 1 (k))-e 2 (H 2 (k)). (23) 

By solving the following set of equations 

()L = and — — j— — - = 0, (24) 



(21) 



dw{+,-) dw(-,+) 
one can express 6\ and 62 as a function of the microscopic parameters 

4 J6l = ln i-«>(+,-)i-t P (-,+) and 209 = h ;-H-.+) (25) 

«»(+,-) uj(-+) 1 -«;(+,-) 

By inverting (|25|) . one gets expressions for u>(+, — ) and w(—, +) as a function of #1 and 62 
l-w(-,+) = e^[l-w{+,-)\ 1 

1 -«;(+,-) = ^cosh(0 2 ) - ^/sinh 2 (0 2 ) + e~ ije ^j (l - e - 4JSl ) _1 e"" 2 . (26) 

In combination with (f2"Tj) . one finally obtains formulas for (iZi(fc)) and (H2(k)) as a function of #1 and 6*2- These 
relations can then be used to estimate the values of 9\ and #2 after measurement of (Hi{k)) and {H.2{k)). 
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IV. GENERAL THEORY 



Our microscopic model is the iV-state Markov chain with parameters p(z) and w(z,y). The only constraints on 
these microscopic parameters are the normalisation conditions (JSJ), © and the detailed balance conditions ©. To 
proceed from this mathematical model to a physical model one has to make a choice for the relevant observables 
Hi(k). Then one can introduce Lagrange multipliers {9i, a, ((z), i](z, y)} and maximise the following function 

j=i zer zer yer 

~E E v&y) W^^f) -p(y)^(y>^)] ■ (27) 

zGryer,y>z 

over the parameters p(z) and w(z,y). Notice the fundamental difference between the constraints that are taken into 
account by the Lagrange multipliers {ftj} and {a, C( z )? ^O 2 , y)}- The former should be determined as a function of the 
model parameters p(z) and w{z,y). The latter are mathematical tools to take into account some basic microscopic 
constraints. These multipliers should be eliminated out of the theory since they are not connected to macroscopic 
observables. 

Before (|27[) can be maximised over p(z) and w(z, y), the parameter dependence of S and (Hi(k)) must be know. 
We already obtained a formula for the entropy as a function of p{z) and w(z,y) only, see expression (fTTj) . In this 
paper, we assume that the observables Hi(k) are linear combinations of the elements of the transition record of the 
Markov chain 

t 

Ewo - EE 6 ^^^ ( 28 ) 

i=i zer yer 

where the elements of the matrix O are some linear combination of the Lagrange multipliers 9{. Taking the ensemble 
average of (|2"5)) and using expression (fTTj|) results in 

i * 

i=i zer yer 

In appendix [5] the optimisation of the function (f2"7| is carried out analytically. One ends up with the following set 
of equations 

ln w( X ,y)w(y,x) = @{XjX) + Q^ y) _ QiXjy) _ e ^ xl 



w(x,x) w(y,y) 
w(x, x 



i n _4^4 = ®(x,x)-e(y,y), (30) 



for all x,y € T. These AT (AT - l)/2 + AT - 1 = (AT - 1)(N + 2)/2 equations together with the A^N - l)/2 detailed 
balance conditions ([8|) and the 1 + N normalisation conditions ([5|) , (J6j) are a closed set of equations for the N + N 2 
microscopic parameters p(x) and w(x,y). To obtain relations for p(x) and w{x,y) as a function of the parameters Oi 
(contained in the elements of the matrix O), one has to invert this set of equations. A part of this inversion can be 
performed generally. Start by choosing an arbitrary state r and rewrite the relations (j8]) and §E§ as follows: 



P«= 1+Efl , p(x)=p(r)^-, VxeV, (31) 
* ' ( w(y, r) / w(a;, r) 



y-v w(r,y) \ r \ t \ w { r i x 

with T' = T\{r}. Then, the remaining detailed balance conditions ([5]) can be rewritten as follows: 

w(r, x)w(x, y)w(y, r) = w(x, r)w(y, x)w(r, y), \fx,y^T'. (32) 

Notice that (f3Tj) expresses the probabilities p(x) as a function of the transition probabilities w(x,y) only. Therefore 
to obtain relations for p(x) and w[x, y) as a function of the parameters 0j one only has to invert the relations (I30p . 
(|32p together with the normalisation conditions ([S]) . This part of the inversion will depend on the particular form of 
the matrix O and has to be performed for every physical model individually. 
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We want to emphasise that our procedure fits in the estimation theory [24j, |25(. In that approach, the average 
values of some observables are used to estimate the values of the model parameters. In the present paper, we make a 
separation between the physical model of a theory and the underlying mathematical model. The latter is the TV-state 
Markov chain while the former model is introduced by identifying some physically relevant observables. Usually, the 
number of microscopic parameters p(x) and w(x, y) of the mathematical model is larger than the number of relevant 
observables Hi(k) of the physical model. By measuring (Hi(k)), only the values of the corresponding parameters 
9i can be estimated. Then, one can calculate an estimation of all the values of p(x) and w(x, y) with the formulas 
obtained in this section. As such, no a priori choice for these parameters is necessary and one obtains the least biased 
values for p{x) and w(x,y) given only the measured information and some basic microscopic constraints. 



V. THERMODYNAMICS 



In statistical mechanics, the starting point to describe a given model is usually the Hamiltonian H (J) with j £ T 
and r the phase space. Then, the standard way of introducing the temperature T is by postulating the Boltzmann- 
Gibbs form exp[—H(j)/T] for the equilibrium distribution. This approach is motivated by the maximum entropy 
principle that we already outlined in the introduction. Indeed, when the Hamiltonian is identified as the only relevant 
observable, expression ((3]) for the equilibrium distribution simplifies to exp[—6iH(J)]. Notice that this corresponds 
to the choice t = 1 and H\{j) — H(j). Using the laws of thermodynamics, one can show that 6\ is indeed the inverse 
temperature. 

In previous sections, we used the maximum entropy principle to obtain the least biased values of the microscopic 
parameters of a mathematical model gives some macroscopic constraints. Since this problem fits in the estimation 
theory, we did not give a thermodynamic interpretation of the Lagrange multipliers 0j. However, such a deeper 
understanding is highly desirable for the application of our theory to physical models like, e.g., the Ising chain. 
Therefore, in this section we study this thermodynamic interpretation in more detail. We first outline briefly the 
concept of exponential families which is very important in this context. Then we illustrate the relation between the 
Lagrange multipliers 9i and the temperature for the 2-state Markov chain. Finally, we show under which conditions 
our technique coincides with the standard approach to introduce thermodynamic parameters into theoretical models. 



A. Curved exponential family 

A distribution with parameters w = [wi, . . . ,w s ] belongs to the i-parameter exponential family when it can be 
written as follows: 

P(j) = c(j) exp (g(w) - J2 ^MfcC?)) , (33) 

where t is the smallest integer for which the exponential form can be obtained, c(j) is a prior probability and G(w) 
is determined by the normalisation condition 

G(w) = - In \J2 ^P ^- J2 & i( w )9i(j?J j ■ (34) 

The family is said to be curved when t > s 26] (s is the dimension of the parameter vector w, see above). The 
special case for which s = t is called a full exponential family. Then, one can interpret the functions 0i(w) as the new 
parameters of the distribution 9 — [8\, . . . , 9 t ]. The Boltzmann-Gibbs form is obtained when also the functions gi(j) 
coincide with the relevant observables Hi(j) 

P(j) = c(j) exp (g(8) - OiHiij^j ■ (35) 

In the next section, we will show that the subtle differences between the curved and the full exponential family are very 
important for the thermodynamic interpretation of the parameters 9. A generalisation of the concept of exponential 
families with applications in the context of nonextensive statistical mechanics is proposed by Naudts [2 a . [27], |28| . 
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B. Example: the 2-state Markov chain 



We studied the 2-state Markov chain already in section IIIII and interpreted this model as a one-dimensional Ising 
chain. We identified two relevant observables -ffi(fc) and H 2 (k), see expressions ([T9")) and (f2T)]) respectively, and used 
(Hi(k)} and (H 2 {k)) as constraints in the maximisation procedure. As a consequence, the matrix O (|28p becomes 



e 



9(1,1) 6(1,2) 
6(2,1) 6(2,2) 



-J9 1 +9 2 J9 1 -9 2 
J9 1 +9 2 -J9x-9 2 



(36) 



Using pop , the parameters 9\ and 9 2 can then be expressed as a function of the microscopic parameters. It is easy 
to check that this procedure results in the same formulas for 9\ and 9 2 that we obtained before (|25|) . as it should be. 
Clearly, (H 2 (k)) is just the magnetisation M of the chain, while (Hi{k)) is usually interpreted as the internal energy 
U of the one-dimensional Ising model. Within this interpretation, the parameters 9\ and 9 2 can be related to the 
temperature T and an external applied field F as follows: 9\ — 1/T and 9 2 = —F/T. One can check this, e.g., by 
showing that the following thermodynamic relations Q hold 

d ~§ = U-FM and § = -M, (37) 

with G the free energy G = U — FM — TS. The final expression for the magnetisation as a function of T and F is 

— = sinh(F/T) (sinh 2 (F/T) + e- 4J / T ) _1/2 . (38) 

This is the well-known result for the one-dimensional Ising model (lp| . We proceed by studying the equilibrium 
distribution that is obtained by our optimisation procedure in order to understand why our final formula for the 
magnetisation (|3"5)) coincides with the standard result. The probability P(k) to observe a Markov chain with certain 
transition record k is proportional to 

P(k) ~ w(+,+) fe(+ ' +) w(+,-) fc(+ ^ ) «;(-,-) fc( -^ ) u;(-,+) fc( -' +) . (39) 

Now we want to express this probability as a function of the relevant variables Hi(k), H 2 {k) and n. Expressions for 
Hi(k) and H 2 (k) as a function of the elements of k are already given in (fT9")l and (|2"D|) . The length of the Ising chain 
n + 1 is just the sum of all the elements of k plus 1, i.e., 

n = fc(+, +) + k{+, -) + -) + fc(- +). (40) 

In this way, we obtain only 3 equations for 4 variables, the 4 elements of the transition record. However notice that 
the difference between the values of k(— , +) and k(+, — ) can only be or 1. Therefore in the thermodynamic limit 
we have an extra constraint for the elements of k 

fc(+,-) = fc(-,+). (41) 

In this way we end up with a closed set of equations (|19l2QI4QI4ip for the 4 elements of k. By solving this set of 
equations for iJi(fc), H 2 (k) and n, we can rewrite expression (|39p as follows: 

P(k) ~ cM-ViHi(k) - 9 2 H 2 {k)], (42) 

see (|25p for the definitions of 9\ and 9 2 . We omitted the dependence of n to obtain (|4"2"1) , because the length of the 
Ising chain is assumed to be constant. Rewriting P{k) in this form makes the thermodynamic interpretation of the 
parameters 9\ and 9 2 as 9\ — 1/T and 9 2 = —F/T immediately clear because expression (|42|) is just the Boltzmann- 
Gibbs distribution exp[— 9iH(k)] 1 with H(k) = Hi(k) + H 2 (k)9 2 /9\. This is indeed the Hamiltonian of the Ising 
chain. We conclude that for this simple example, our technique to introduce the thermodynamic temperature in a 
statistical model coincides with the standard approach. As a consequence, it is no surprise that our expression for 
the magnetisation ([38]) is equal to the well-known result for the one-dimensional Ising model. 

Notice that ignoring equation (|4ip in this procedure results in an extra contribution to expression (I42p 

P(k) ~ vq>[-9iHi(k) - 9 2 H 2 {k) - 9 a H 3 (k)}, (43) 

with 

2T a (Jfe) = *(+,-) -fc(-+) and 9 3 = \ In g=iz|g=|±| . (44) 
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Observe that the distribution ([43]) is a member of the curved exponential family. Indeed, the 2 independent parameters 
are w = [w(+, — ), w(— , +)]. However, in order to rewrite P{k) in an exponential form, one needs 3 functions 
= [01,02,03] of these parameters instead of 2. Therefore, the distribution (|43| belongs to the curved exponential 
family while the distribution (|42[) is a member of the full exponential family. As a consequence, the interpretation of 
the parameters of the distribution (|43[) is not immediately clear. As we mentioned before, the difference between the 
values of k(— , +) and k(+, — ) can only be or 1. So for this particular example, the difference between the full and 
curved exponential family only occurs for finite systems. However, the problem is more general. In this paper, we 
study mathematical models with an arbitrary number of microscopic parameters. The number of physically relevant 
observables ifj(fc) is usually a lot smaller. Therefore, it is not obvious whether the distribution P(k) belongs to the 
full or curved exponential family in the variables Hi(k). Or equivalently, it is not obvious whether it is possible to 
rewrite the probability P(k) in the Boltzmann-Gibbs form. We examine this question for the iV-state Markov chain 
in the next section. 



C. Boltzmann-Gibbs distribution 



The conditional probability P(k;xo) to observe a Markov chain with certain transition record k given the initial 
condition xq € T can be written as (Ti~2"|) . We also derived relations between the elements of the transition record (fT4"|) 
that must be satisfied in order to obtain a non- vanishing value for P(k; xq). The dependence of P(k; xq) on the initial 
condition xq is unimportant for large Markov chains. Therefore, we ignore the dependence of xq in the remaining part 
of this section and write the probability P(k) to observe a Markov chain with certain transition record k as follows: 

P(k) = c(fc)exp I ^2^2k(x,y)lnw(x,y) J := c(k) exp (T(fc)) . (45) 

When finite size effects are ignored, expression (fl"4"|) simplifies to 

X>(r,y) = $>(2/,r), Vreh (46) 
y er v er 

Notice that these relations are the generalisation of expression (l4"Tj) . In this paper we have assumed that the detailed 
balance conditions ([8]) hold and derived relations (|30[) between w(x,y) using the maximum entropy principle. The 
aim of this section is to examine whether the conditions (|4"6"|) together with ([5]) and (f3"0|) are sufficient to rewrite P{k) 
as the Boltzmann-Gibbs distribution (|3"5|) 

P(k) ~ c(k) exp ( - QiHiik)) = c(k) exp [ -££ k(x, y)6(x, y) j . (47) 
V *=i / \ xer y er ) 

See expression (|2"8")) for the definition of the matrix O. Notice that we omitted the dependence of the normalisation 
G(0). By using the conditions ©, flM]) and ([301), we rewrite T(fc) (gSJ) as follows: 

w(x,y) + r^^^i,!/) \nw(x,y) 
xeryer xeryer 

= \ y } ln y } + \ fc ( a; ' y "> ln x "> 44 



xer yer 

= ■^^}2 , }2k(x,y)[Q(x,x) + Q(y,y) - Q(x,y) - Q(y,x) + lnw(x,x)w(y,y)} . (48) 
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Then, we use (|46| and ([30]) again together with /c(x, y) = ?i to prove the following equality 

iEE^^'^^ ^'^^ ^'^^ 111 ^^'^^^'^)] = ri [ ( r > r ) +lnw(r,r)] , (49) 



2 

xer yer 
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with r an arbitrary state. Since we assumed n to be constant, this term can be absorbed in the normalisation of the 
distribution P(k). Therefore, we end up with the following expression 

p(k) ~ c (k) exp ( ~ E y) t 9 ^ y) + e ^ x )i I ■ ( 5 °) 

The distribution P{k) is only of the Boltzmann-Gibbs form when the following condition holds (compare (|50|) with 

m) 

= ^^fc(a;,y)[e( a ;,y)-e( 2 /, a ;)], (51) 
x<ET yer 

or equivalently 

= ^^[%!,)-%i)]e(^). (52) 

A sufficient condition for the equality (f5Tj) to hold is obviously 0(x, y) = 9(y, a:) for all x, y € T. However, the results 
of previous section show that this constraint is to restrictive. Indeed, we showed for a simple example that one can 
rewrite the distribution P(k) in the Boltzmann-Gibbs form, without the matrix ()36j) being symmetric. The crucial 
observation to obtain this result was that following constraint, k(+, — ) = k(— , +), is fulfilled for the 2-state Markov 
chain. For TV-state Markov chains, the latter equality can be generalised to (|46|) . We proceed by eliminating some of 
the elements of the transition record out of expression (|5l"j) by using the conditions (|46|) . Chose an arbitrary state r 
and replace in expression (f5Tj) . k(x, r) by 

k(x,r) = k(r,x)+^2[k(y,x)-k(x,y)}, (53) 
with r' = r\{r}. This results in the following condition 

xer' j/er' 

A sufficient condition for this equality to hold is 

0(r,a;)+e(x,|/) + e(i/,r) = @(x,r) + @(y,x) + Q{r,y). (55) 

Notice that this is a similar constraint to the detailed balance condition for the transition probabilities (|3"2"|) . This 
derivation does not depend on the arbitrary chosen state r. As such, expression (|55|) should hold for all r,x,y G T. 
For the examples studied in this paper, condition ([55]) will always be satisfied. As a consequence, the relation between 
the Lagrange multipliers 9i (contained in the matrix O) and the thermodynamic parameters is immediately clear. 
Therefore, in the remaining part of this paper, we will omit the substep of explicitly checking thermodynamic relations 
like (|37p during our analysis. 



VI. EXAMPLE: THE 3-STATE MARKOV CHAIN 



In this section we study the 3-state Markov chain. The three states are denoted 1, 2 and 3. The different parameters 
of the Markovian model are 



p(l),p(2), p(3) and w(l, 1), w(l, 2), w(l, 3), w(2, l),w(2, 2), w(2, 3),w(3, 1), w(3, 2),w(3, 3). 

However, the number of independent parameters is reduced by 4 because of the normalisation conditions ([5]), ([6]). 
The detailed balance conditions ((HJ further reduce the number of independent parameters by 3. We conclude that 
this microscopic model contains 5 independent parameters. In section |H] we introduced the transition record k. For 
this example, the matrix k contains 9 elements 



fe(l,l) fc(l,2) k(l,S) 
k(2,l) k(2,2) fc(2,3) 
fc(3,l) fc(3,2) fe(3,3) 



(56) 
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In the next two sections we study two physical models that are contained in this 3-state Markov chain. The relevant 
observables of these two models are different and, as such, the elements of the matrix 8 are not equal. As a 
consequence, the constraints (|30|) that relate the thermodynamic parameters (contained in the matrix 8) to the 
microscopic parameters w(x,y) will be different for the two physical models. However, the relations ([51"]) . (j3"2")) and 
([6]) are the same because they only depend on the mathematical 3-state model and not on the particular choice of the 
relevant observables. Therefore we write out the formulas (I31|) . (|32|1 and (J6j> here, before we proceed with studying 
the expressions (13TH) in the next two sections. We choose r = 1 in expressions (|3"Tj) and 

p( w(2,l)w(3,l) 



w{2, l)w(3, 1) + 10(1, 2)10(3, 1) + w(2, l)w(l, 3) ' 

*< 2 > = ^%r^ p{3) - ^M p ^ (57) 

10(2,1) w(3,l) 
w(3, l)w(l, 2)w(2, 3) = w(l, 3)w(2, l)w(3, 2), (58) 

and write out the normalisation conditions {21 for the transition probabilities 

10(1, 1) = 1 - 10(1, 2) - 10(1, 3), 10(2, 2) = 1 - 10(2, 1) - 10(2, 3), 

t0(3,3) = 1 -10(3,1) -10(3,2). (59) 



Potts model 



The 3-state Markov chain can be interpreted as a one-dimensional Potts model [TJ [T3] ■ This system corresponds 
to a chain of n + 1 spins. Contrary to the Ising model, the spin variables <ii are vectors with unit length that can 
point in 3 directions specified by the angles 2g7r/3 with q = 0, 1, 2. Two relevant observables are 



H\{a) = — J Gj ■ <r i+ i and H 2 (a) = 1 • Oj 



(60) 



i=0 



i=0 



where J is a constant and 1 is a unit vector that points in one of the spin directions. Clearly, (H 2 (a)} is just the 
magnetisation M of the chain along the direction of 1, while (H\(a)) is usually interpreted as the internal energy U 
of the one-dimensional Potts model. The three states of the Markov chain 1,2,3 correspond to the three different 
spin directions. The contribution to -Hi(cr) is — J or J/2 depending on whether Ui = er,;+i or cr, =/= respectively. 
Therefore, one can express -Hi(c) as a function of the elements of the transition record k as follows: 



H^k) = -J[k(l, 1) + k{2, 2) + fe(3, 3)] 

+ ji [fc(l, 2) + fc(l, 3) + k(2, 1) + k(2, 3) + fc(3, 1) + fc(3, 2)]. 



(61) 



In order to obtain a similar expression for H 2 (<r) , we chose arbitrarily the direction of 1 along the state with label 1 
H 2 (k) = fc(l, 1) + fc(2, 1) + fc(3, 1) - -[k(l, 2) + k(2, 2) + fc(3, 2) + fc(l, 3) + fc(2, 3) + fc(3, 3)]. (62) 



Analogous to the example of the Ising model, see section IIII| we ignored the unimportant finite size contribution 
of the initial spin oo to obtain H 2 {k) from H 2 (a). We use (-ffi(fc)) and (H 2 (k)} as constraints in the maximisation 
procedure. As a consequence, the matrix 8 (I28|) becomes 



8 = 



8(1,1) 8(1,2) 8(1,3) 
8(2,1) 8(2,2) 8(2,3) 
8(3,1) 8(3,2) 8(3,3) 



-2 J0i + 26» 2 
J0i + 20 2 
J0i + 20 2 



J0i - 

-2J0! 

J01 - 



J01 - 02 
J0 t ~ 02 
-2J0i - 2 



(63) 



Using (|30p. the parameters B\ and 02 can then be expressed as a function of the microscopic parameters as follows: 



3J0i = In 



In 



10(1,1)^(3,3) , i0(2,2)t0(3,3) 
— m 



^02 = m^M 
2 10(1,1 



10(1,1)10(2,2) 

10(1,2) 10(2,1) ~ 111 10(1,3) 10(3,1) ~ 111 10(2, 3) 10(3, 2) ' 
10(3,3) 



= In 



10(1,1) 



(64) 
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FIG. 1: Plot of the magnetisation of the one-dimensional Blume-Emery-Griffiths model as a function of the external applied 
field at constant temperature 1/T = 20. The values of the constants of Hi (|68[) are K = 0, J = —1 and A = 0; 0.5; 1 for the 
dotted, the solid, the dashed line respectively. 



Together with (|58|) and (|59|) these expressions form a closed set of equations that relate the parameters 0\ and 02 to 
the microscopic parameters w(x,y). In appendix [Bl this set is inverted analytically. We proceed by writing out the 
ensemble average of H 2 (k) with fit)]) . (|57| and (|59| 



(gaW) = 1 , 3 w(2,l) 

n 2 2w(2,l) +2tu(l,2)' 1 j 

In combination with the results of appendix IE) one then obtains a formula for (H2(k)) as a function of 0\ and 02- 

These parameters can be related to the temperature T and an external applied field F as follows: 6\ = 1/T and 
62 = —F/T. The final expression for the magnetisation M = (H2(k)) as a function of T and F is 

M 1 3 e 3J/2T _, 1 + e 3J/2T\ e -3F/2T 

— = A + A I —o = ( 66 ) 



^[ e 3J/2T _ + £ 3J/2T) e -3F/2T] 2 + gg- 



An identical expression is obtained in [12j . The authors assume that the equilibrium distribution is of the Boltzmann- 
Gibbs form and solve the one-dimensional Potts model with the technique of the transfer-matrix. Since the Hamilto- 
nian of this model satisfies (|55|) . the resulting equilibrium distribution of our approach is also of the Boltzmann-Gibbs 
form. That's the reason why the final expressions for M as a function of T and F of the two different approaches 
coincide. 



B. Blume-Emery-Griffiths model 

The 3-state Markov chain can be interpreted as a one-dimensional Blume-Emery-Griffiths model [H, [l|J|. This 
system corresponds to a chain of n + 1 spins. The spin variables cr, are scalars that can take on three values +1, 0,-1. 
Two relevant observables are 

n— 1 n — 1 n n 

i=0 i=0 i=0 i=0 

where J,K,A are constants. Clearly, (^(c)} is just the magnetisation M of the chain, while (Hi(a)} is usually 
interpreted as the internal energy U of the one-dimensional Blume-Emery-Griffiths model. The three states of the 
Markov chain 1, 2, 3 correspond to the spin values +1,0, —1 respectively. Within this interpretation, one can express 
H\(a) and i?2(f ) as a function of the elements of the transition record k as follows: 



ffi(fe) = -J[k(l, 1) + fc(3, 3) - fc(l, 3) - fe(3, 1)] - K[k(l, 1) + fc(3, 3) + fc(l,3) + k(3, 1)] 
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H 2 (k) 



+A[k(l, 1) + k(2, 1) + fc(3, 1) + fc(l, 3) + fc(2, 3) + fc(3, 3)], 
fc(l, 1) + fc(2, 1) + fc(3, 1) - [fc(l, 3) + k(2, 3) + fc(3, 3)]. 



(68) 
(69) 



Analogous to previous examples, we ignored the contribution of A<7q to obtain Hx(k) from H\(a) and the contribution 
of (To to obtain H2(k) from H 2 (a). We use (Hi(k)) and (^(fc)) as constraints in the maximisation procedure. As a 
consequence, the matrix 9 (|28|) becomes 



6 



9(1,1) 9(1,2) 9(1,3) 
9(2,1) 9(2,2) 9(2,3) 
9(3,1) 9(3,2) 9(3,3) 



h(-J-K + A) + 6 2 0i(J-# + A) -0 2 

0iA + 2 x A-0 2 

e l {J-K + A) + 9 2 0i(-J-K + A) -6 2 



(70) 



Using (|30[) . the parameters 6\ and 02 can then be expressed as a function of the microscopic parameters as follows: 



W + K) 
-40x J 



In 



In 



-9 X {J + K- A) + 2 = In 
-9i(J + K- A) -0 2 = In 



w(l,2) w(2, 1) 

u)(l,l)t«(2,2) 
io(l,3) w(3, 1) 

w(l,l) w(3,3)' 
^(2,2) 

to(l, 1) ' 
w(2,2) 

to(3,3)' 



In 



to(2,3) w(3,2) 
w{2,2) w(3,3)' 



(71) 



Together with (f58|) and (l59|) these expressions form a closed set of equations that relate the parameters 0\ and 02 
to the microscopic parameters w(x,y). In appendix [Cl this set is inverted analytically. As in previous example, the 
parameters 0i and 02 are related to the temperature T and an external applied field F as follows: 0i = 1/T and 
02 = —F/T. We proceed by writing out the ensemble average of H 2 {k) (|69[) with (fT0| . (|57| and (|59|) 



w(3, l)w(2, 1) - w(2,l)w(l,3) 

u>(3, l)w(2, 1) + w(l, 3)w(2, 1) + w(l, 2)w(3, 1) ' 



(72) 



In combination with the results of appendix [Cl one finally obtains an expression for the magnetisation M = (i?2(fc)) as 
a function of T and F. A plot of the magnetisation as a function of the external applied field at constant temperature 
0! = 1/T = 20 is shown in figure [1] for the following values of the constants of Hi(k) (f6"5)) K = 0, J = — 1 and 
A = 0; 0.5; 1. It is known that multiple plateaus show up is this curve depending on the value of A 0, [2i|. This 
interesting behaviour can also be observed in figure [1] 



VII. DISCUSSION 



In this paper, we present a general procedure to estimate parameters in Markovian models. The Markov chain is 
a mathematical model that is defined by initial probabilities p(z) and transition probabilities w(z,y). We interpret 
p(z) and w{z,y) as the microscopic parameters of the Markovian model. Then, relations between p(z),w(z,y) and 
some relevant control parameters 0j are determined with the maximum entropy principle. Finally, one ends up with 
formulas that express the average values of the relevant observables Hi(k) as a function of the corresponding control 
parameters 0, only. These expressions can be used to estimate the values of 0i after the measurement of (Hi(k)). We 
want to stress that the dependence on the microscopic parameters is completely eliminated out of the theory. This 
means that no a priori choice for the values of p(x) or w(x, y) is necessary. This is important because the values of 
these parameters are not measurable. 

We made a clear separation between the physical model of a theory and the underlying mathematical model. The 
latter is the iV-state Markov chain while the former model is introduced by identifying some relevant observables. As 
such, different physical models can be contained in one type of Markov chain. This is illustrated in section IVII where 
we examined two different physical models that are contained in the 3-state Markov chain. We showed that is possible 
to perform the aforementioned optimisation procedure in full generality for the iV-state Markov chain. This results in 
relations Q30p between the microscopic parameters of the mathematical model and some relevant control parameters. 
These formulas are the main result of this paper. 

In section IV C I we studied under which conditions the equilibrium distribution of our approach is of the Boltzmann- 
Gibbs form. Gbtaining this type of equilibrium distribution is advantageous because a thermodynamic interpretation 
of the control parameters is obvious in that case. We derived a sufficient condition that is satisfied for all the examples 
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studied in this paper. As such, the final formulas for the average values of the relevant observables as a function of 
the thermodynamic parameters that are obtained in this paper have been studied before. The general procedure to 
obtain these formulas is the novel contribution of this paper. Notice that further generalisations of our technique 
are still possible. We assumed that the relevant observables are linear combinations of the elements of the transition 
record of the Markov chain. In [3(|, this condition is lifted. In that paper, the specific example of the 2-state Markov 
chain with a mean-field Hamiltonian is studied with the same technique as described in the present paper. It is an 
interesting topic for further research to examine the effect of allowing mean-field Hamiltonians in the theory for the 
general TV-state Markov chain. We also assumed that all the transitions are allowed (w(x,y) ^ for all x, y G T). 
Lifting this assumption will usually cause the violation of the detailed balance condition. The generalisation of the 
results reported in the present paper to non-equilibrium steady states is currently under study. Notice that allowing 
for a vanishing transition probability w{x,y) will not cause the violation of the detailed balance condition when the 
transition y — > x is also not allowed. The specific example of a 6-state Markov chain with that property is studied in 
Q. Throughout the present paper, we ignored finite size effects. The technical consequences of taking these effects 
into account are already thoroughly examined for the 2-state Markov chain in 0, [l!| . 
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APPENDIX A 



In this appendix we maximise the function 

-C = -^2^2p(z)w(z,y)\nw{z 7 y) -^2^2<d(z 7 y)p(z)w(z,y) - a^2p(z) 
zer yer zer yer zer 

-^2^( z )^2 w ( z ^y) V{z,y)[p(z)w(z,y) -p(y)w(y,z)], (Al) 

zer yer zer yer,y>z 

over the parameters p(z) and w{z,y). Therefore, we set the first derivative of C with respect to these parameters 
equal to zero. The resulting equations for differentiating with respect to w(u, u) (|A2[) . p(u) (|A3|) . w(u, v) (j A4|l . w(v, u) 
(|A5jl with v > u are 

((u) = -p(u)[l + lnw(u,u) + @(u,u)], (A2) 
= w(u, y) \nw(u, y) + 6(~u, y)w(u, y) + a 

+ V( u ,y)w(u,y) - ^2 v(y, u ) w (u,y), (A3) 

yer,y>u yer,y<u 

= p(u) [1 + lnw(u, v)] + ®(u, v)p(u) + C(u) + r)(u, v)p(u), (A4) 
= p(v) [1 + lnw(v, u)] + Q(v,u)p(v) + £(v) — r](u, v)p(v). (A5) 

One can simplify the expressions (|A4|) and (|A5j) with the use of the formula for (|A2|) 

7/1(7/ 7/1 

= In 1 { + @(u,v)-e(u,u)+r)(u,v), (A6) 
w(u, u) 

= In W y' U ( + 80, u) - Q(v, v) - rj(u, v). (A7) 

W\V, V) 

Combining these two equations results in 

e(u,u) + Q(v,v)-Q(u,v)-Q(v,u) = In w ( u > v \ W M (A8) 

w(u, u) w(v, v) 



We proceed by rewriting expression (|A3[) as follows: 

= w(u, u) \biw(u, u) + G(it, u)] + a 
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- ^2 w(u,y)[\nw{u,y) + Q(u,y)+i](u,y)} 

y£T,y>u 

^2 u>(u,y)[lnw(u,y) + Q(u ) y) -rj(y,u)}. (A9) 

yer,y<u 



Then, we use (IA6I) and (IA7I) to transform (IA9I) further to 



= w(u, u) [In w{u, u) + 0(it, it)] + a 

+ y) [In w(u, m) + 9(m, u)] 

j/er,j/>M 

+ y) [lnw(u, u) + 0(m, u)] 

yeT,y<u 

-a = lnw(u,u) + Q(u,u). (A10) 

The latter equation is valid for all u €T. The parameter a can then be eliminated, by combining these equations two 
by two 

e(u,u)-Q{v,v) = ln W ) V,V { . (All) 

APPENDIX B 

In this appendix we invert the set of equations (|64|). (|58|) . ([59| 



3Jfll = 1 - w(l, 2) - w(l, 3) 1 - w(2, 1) - w(2, 3) 
w(l,2) t«(2,l) 
a, 2 = 1- W (2,1)- W (2,3) 
1-^(1,2) -io(l,3)' 
1 1 - tu(2, 1) - iu(2, 3) 1 1 - w(3, 1) - tu(3, 2) 



w(l,2) w(2,l) w(l,3) tw(3,l) 

1 -tw(l,2) -w(l,3) 1 _ 1 1 - u>(3, 1) - u>(3, 2) 



(Bl) 
(B2) 
(B3) 

w(l,2) «J(2,1) w(2,3) w(3,2) ^ 

1 - tu(2, 1) - w(2, 3) = l-iu(3,l)-io(3,2), (B5) 
w(3,l)w(l,2)w(2,3) = w(l,3)w(2,l)w(3,2), (B6) 

to obtain formulas in closed form for the microscopic parameters w(x,y) as a function of 6\ and 02 only. Equations 
(jB3|) . (|B5|l and (|B6j) can be simplified to 

iu(l, 3) =it;(l,2), w(3,l) =iy(2,l) and iw(3,2) = w(2,3). (B7) 

Notice that this restricts the values of io(l,2) and u>(l,3) to the interval [0..1/2], because of the normalisation 
condition. Using (|B7|) . equation (|F34[1 can be rewritten as follows: 



n 2 s = ^(2,3) 2 

Wl ' ' 2w(2,3) 2 + u;(2,l)[l- W (2,l)- U ;(2,3)]' 1 ' 

Inserting (|B7[) and (|B8|) into equations (|B1[) and (|B2[) results in two equations in the variables w(2, 3) and iu(2, 1). 
By inverting these two equations 

u>(2,l) = l-w(2,3) (l + ei jei ) , 

e l^+ (l + e f^i] e |e 2 

^(2,3) = / r-r :„ ' , (B9) 







_ gfj^o 




2 








e |e 2 


4 





2 2- l + e§ je Ae§ Jf> i 



(l + ei^) 

one finally obtains a closed chain of equations for the transition probabilities. 
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APPENDIX C 



In this appendix we invert the set of equations (|7ip . (|58|) . (j59j) . We first introduce a shorthand notation 

m = e -o 1 (J+K) i rj2 = e -^J > Jte = e -e 1 ( J +jr-A)+fl B) ^^(j+jc-Aj-fc (C1) 

and the substitution 

y = w{l,2) w(2, 1) w(2,3) 

1 w(l,l)' 2 u>(2,2)' 3 u>(2,2)' 

_ w(3,2) Ml, 3) w(3,l) 

X4 -^(3T3)' l5 -MMj' X6 "^3)' (C2) 

to obtain the following expressions for the normalisation conditions (|59[) 

k;(1,1) = (1 + X 1 +X 5 )- 1 ! w(2,2) = (I + X2 + X3)- 1 , u>(3, 3) = (1 + X 4 + X & )~ 1 , (C3) 

and the equations (|7T|) . Ij55]l 

771 = XiX 2 , 771 = X 3 X 4 , 7/| = X S X 6 , XiX 3 X 6 = X2X4X5, (C4) 
X 5 = 773(1 + X 2 +X 3 )-1-X 1 , X i =r) i (l + X 2 +X 3 )-1-X 6 . (C5) 



We proceed by rewriting (1C4|) as follows: 

t? 1 = X 1 X 2 , 7 ?1 =X 3 X 4 , 772X3 = X 2 X 5 , 772X3 = X 3 X 6 . (C6) 
Then, we use the expressions (|C5I) for X4 and X5 to transform (|C6[) further to 



X x = ^-, X 6 = 774(1 + X 2 +A 3 )-1-^, (C7) 

772X3 =X 2 773(1 + X2 +X 3 )-X2 - 7?i, 772X2 =X 3 774(1 + X2 +X 3 )-X3 - 7?!. (C8) 

Finally, we rewrite (|C8[) as follows: 

X 3 = ^3(1 + X 2 )-X 2 -77 1; 

7/2 - X 2 T7 3 

(X 2 77 2 + 77i)(X 2 ?7 3 -?7 2 ) 2 = [X 2 77 3 (l+X 2 )-(X 2 +77 1 )]x 

[774772(1 + X 2 ) - 774 (X 2 + 77O + X 2 77 3 - 773] , (CIO) 

to obtain a closed chain of equations for the variables Xi with i = 1 ... 6. Expression (jClOj) is a cubic equation in 
the variable Xi which can have 3 real solutions. However, it is well known that one-dimensional systems with short 
range interactions do not exhibit phase transitions when the equilibrium distribution is of the Boltzmann-Gibbs form. 
Therefore, only one of the solutions of the cubic equation is physically meaningful. The other solutions are complex 
or result in values for some of the transition probabilities outside the interval [0, 1]. 
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